MENLI: Robust Evaluation Metrics from Natural Language Inference

Abstract Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating information correctness. We argue that this stems (in part) from the fact they models of semantic similarity. In contrast, we develop based Natural Language Inference (NLI), which deem a more appropriate modeling. design preference-based attack framework and show our NLI much robust attacks than recent metrics. On benchmarks, outperform existing summarization metrics, below SOTA MT However, when combining with obtain both higher robustness (15%–30%) quality as measured (+5% 30%).